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Abstract 

The objective and contribution of the research 
presented in this paper is to provide a Multi-Mode 
Manipulator Display System (MMDS) to assist a 
human operator with the control of remote manipulator 
systems. Such systems include space based 
manipulators such as the space shuttle remote 
manipulator system (SRMS) and future ground 
controlled teleopera ted and telescience space systems. 
The MMDS contains a number of display modes and 
submodes which display position control cues position 
data in graphical formats, based primarily on 
manipulator position and joint angle data. Therefore 
the MMDS is not dependent on visual information for 
input and can assist the operator especially when visual 
feedback is inadequate. This paper provides 
descriptions of the new modes and experiment results to 
date. 

1. Introduction 

Manual control of a remote manipulator can be a 
difficult task due, in part, to a lack of useful feedback to 
the operator on the position of the manipulator with 
respect to its desired position, destination, or target 
object to be manipulated. For example, to control many 
remote manipulator systems, including the space shuttle 
remote manipulator system (SRMS), the operator relies 
largely on visual feedback from direct views through 
windows and indirect views from cameras. However, 
the visual information can be insufficient in providing 
the operator with adequate cues, due to obstructions, 
poor viewing angles, camera failures, or problems with 
resolution or camera control. 

The Multi-Mode Manipulator Display System 
(MMDS) is being developed by MDA to alleviate some 
of these difficulties. The current design of the MMDS 
consists of two major modes: 1) the Manipulator 
Position Display (MPD) mode, and 2) the Joint Angle 
Display (JAD) mode. At the time of the writing of this 
paper, the MPD mode has undergone testing and is 


further along in the development cycle than the JAD 
mode which is in its initial development 

2. Manipulat or Position Display (MPD) Mode 

The Manipulator Position Display mode consists of 
two sub-modes: 1) Rotational/Translational (R/T) 
Submode, and 2) MPD Pilot Submode. The two 
submodes of the MPD were designed to help alleviate 
the problems associated with poor visual feedback 
I caused by obstructions, poor viewing angles, poor 
; resolution, camera control, or camera failure. This can 
be done because the MPD does not rely on visually 
obtained information as a source of input, but rather on 
six degree of freedom position information data from 
the manipulator system sensors (for example, joint 
position encoders). 

Further, with the MPD displays, six degree of 
freedom position cues are displayed to the operator in a 
graphical format. The MPD displays the six degree of 
freedom cues concurrently. In addition, the MPD's 
algorithm performs the necessary calculations and 
provides the operator with "fly-from" or "fly-to” cues 
that alleviate the burden of calculating the appropriate 
system inputs from the operator. 1 

The MPD needs to know the current and desired 
(or target) positions. The current position of the 
manipulator arm can be obtained through real time 
position data from the manipulator arm in six degrees of 
freedom. The desired position of the arm in six degrees 
of freedom needs to be identified and entered into the 
MPD program. With this knowledge, the MPD 
displays can present the deviation or error that exists in 
each degree of freedom to the operator in an easy to use 
format. The MPD displays not only have applications 
for the SRMS, but also for other human-machine 
applications (aircraft, deep sea manipulators, nuclear 
environment, etc.) which require the operator to control 
multi-degree of freedom systems under limited viewing 
conditions when desired target points are known. 
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The experiments conducted with the two submodes 
of the MPD mode showed that using either submode 
significantly improved operator performance (by 25 to 
33%) over performing the same manipulation tasks 
without the use of the MPD submodes. 2 ' 3 

2.1 Rotational/Translational Submode 

Figure 1. shows the format of the 
Rotational/Translational (R/T) Submode of the 
Manipulator Position Display mode. 3 The 
Rotational/Translational Submode separates the 
rotational and translational cues to be represented by the 
motion of two separate objects. This submode was 
designed so that one object on the display would 
correlate exclusively to the translational inputs on the 
hand controllers, while the second object would 
correlate exclusively to rotational inputs on the hand 
controllers. 
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Fig. 1. MPD Rotation/Translation Submode Format 

The line in the center with the three tick marks in 
Fig. 1 is stationary and acts as the reference line. The 
operator drives the translational cues using the square 
with the tick marks shown in Fig.l. Deviation in Z- 
translation is depicted by the square being above or 
below the reference line, while Y-translation deviation 
is shown by the square being to the left or right of the 
center of the reference line. For X-translation, the 
operator relies on the size of the square relative to the 
length of the reference line. For rotational cues the 
operator would look to the circular object shown in Fig. 
1. The position of the circle with respect to the 
reference line provided the rotational deviation 
information to the operator. If the circle is above or 
below the reference line, a deviation in pitch exists. A 
deviation in yaw is depicted by the circle being to the 


left or right of the center of the reference line. Roll 
cues are provided by the orientation of the extended line 
running through the center of the circle and the shorter 
line in the center of the circle. If those lines are tilted to 
the left or to the right, then a deviation in roll exists. 

2.2 MPD Pilot Submode 

The format of the MPD Pilot Submode of the 
Manipulator Position Display is shown in Fig. 2. 4 The 
MPD Pilot Submode got its name because it utilizes 
cues, such as a yaw ball and a pitch horizon line, similar 
to those found in aircraft. The line in the center with 
the three tick marks is stationary and acts as the 
reference line. The operator drives five of the six 
position/orientation cues to that reference line, all 
except the yaw cue which is shown separately at the 
bottom of the display. 
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Fig. 2. MPD Pilot Submode Format 

All deviations in the translational degrees of 
freedom are displayed by the circle with the crosshairs 
inside of it. If the crosshairs are to the left or right of 
the center of the reference line, a deviation in Y- 
translation exists. A deviation in Z-translation is 
depicted with the circle and crosshairs being either 
above or below the reference line. Errors in X- 
translation is depicted as a size difference between the 
circle with crosshairs and the length of the reference 
line. For rotational cues the operator would look to the 
yaw ball at the bottom of the display, the horizontal 
pitch line (shown just below the reference line in Fig. 
2), and the orientation of the crosshairs in the ball for 
roll information. The error in the yaw degree of 
freedom are shown by the yaw ball in Fig. 2 being to 
the left or right of center. Pitch error is shown by the 
horizontal pitch line being above or below the reference 
line. For roll cues the operator uses the orientation of 
the crosshairs inside the ball. 
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In addition, for both of the submodes discussed, the 
operator is provided with a digital readout of the 
deviations in each of the six degrees of freedom. This 
digital readout can be seen in the upper left hand comer 
of Figs. 1 and 2, and is helpful in the final stages of a 
task to ensure that the deviations are within the desired 
limits (i.e. close to zero). 

Both submodes also contain two bar graphs on 
either side. The bar graph shown on the left of Figs. 1 
and 2 provides rate information, and the bar graph on 
the right of Figs. 1 and 2 provides the absolute closure 
distance between the current manipulator position and 
the desired manipulator position. This information can 
be particularly helpful to control the rate of movement 
based on the distance from the target location. For 
example, if the manipulator were far from the target 
location the operator would probably want to moving 
faster than if the manipulator was very close to the 
target location. 5,6 

2.3 Experimental Resul ts with the MPD Mode 

To quantify the effectiveness of the two submodes 
of the MPD described in the previous sections, 
experiments with human operators were conducted. 
The MPD display submodes were presented to four 
trained and experienced test subjects on a GRID 1660 
laptop computer. A space shuttle SRMS task was 
simulated using the Manipulator Analysis - Graphic, 
Interactive, Kinematic (MAGIK) simulation system 
which runs on Silicon Graphics computers. The task 
was a space station assembly task, which focused on the 
ins tallatio n of a Pressurized Mating Adapter (PMA) to a 
space station module. 

Three experimental conditions were tested: 1) 
performing the task with the aid of the 
Rotational/Translational Submode of the Manipulator 
Position Display, 2) performing the task with the aid of 
the MPD Pilot Submode of the Manipulator Position 
Display, and 3) performing the task without the aid of 
the MPD display mode. For all three experimental 
conditions the operators were given the clearest 
available camera view of the task (simulated by the 
MAGIK system ) 1 . In addition, the operators were also 
given a digital readout of the position of the 
manipulator in each degree of freedom through a 
simulation of the SRMS display panel. During the 
experimental condition of performing the task without 
an MPD display, this digital position information was 
critical for the final steps of the task when the camera 
view became less helpful. 

Each test subject completed training for performing 
the task without the MPD and with each submode of the 
MPD. Training ended when the test subject's 
performance times reached steady values and learning 


curves flattened. Three separate experimental sessions 
were conducted for each subject. During one 
experimental session, the subject performed the task 
without the MPD display , in another session the 
subject performed the task with the MPD Pilot 
Submode, and in a third session the subject performed 
the task with the Rotational/Translational Submode. At 
the start of each experimental session, each subject was 
given warm-up trials and then six to ten data trials were 
conducted. The subject could end an experimental trial 
when the deviation in each translational degree of 
freedom was less than 1 inch, and the deviation in each 
rotational degree of freedom was less than 0.5 degrees. 

The mean task times for performing the tasks under 
the three experimental conditions are shown in table 1 . 

Pilot Submode No MPD R/T Submodfi 

3.9 min 5.2 min 3.5 min 

Table 1. Mean task times. 

Fig. 3 shows the total average task times calculated 
across all of the four test subjects. The total mean task 
time averaged for all four test subjects was 3.9 minutes 
with a mean standard error of 0.12 minutes when using 
the MPD Pilot Submode , 5.17 minutes with a mean 
standard error of 0.19 minutes when not using the 
MPD, and 3.54 minutes with a mean standard error of 
0.14 minutes when using the Rotational/Translational 
Submode. Therefore, the Rotational/Translational 
Submode provided an average improvement of 
approximately 33% while the MPD Pilot Submode 
provided an average improvement of approximately 
25%. These results were statistically significant to the 
99% confidence level. 


5.5 t 



with without with 

Pilot MPD R/T 

Submode Submode 

Experimental Condition 

Fig. 3. Average of experimental results 
for all test subjects. 


3 


341 


A statistical analysis with a series of paired t- tests 
showed that using the MPD Pilot Submode significantly 
improved operator performance at the 99% confidence 
level (t(30)=7.44, p<0.01). A series of t-tests were also 
conducted to determine the statistical significance of 
using the Rotational/Translational Submode versus not 
using the MPD. As was found with the MPD Pilot 
Submode, the Rotational/Translational Submode 
significantly improved operator performance at the 99% 
confidence level (t(30)=7.41, p<0.01). The statistical 
analysis of the results of using the Pilot Submode 
versus using the Rotational/Translational Submode 
produced differing conclusions based upon individual 
test subject performance. Test subjects #1 and #3 
performed significantly better with the 
Rotational/Translational Submode than with the Pilot 
Submode. However, for test subjects #2 and #4, there 
was no significant performance difference between 
using the Rotational/Translational Submode versus 
using the Pilot Submode. The total average over all 4 
test subjects did show a significant performance 
advantage with the Rotational/Translational Submode 
over the MPD Pilot Submode (t30)=2.06. p<0.05). 


than the other lines. In turn, when the POR of the 
manipulator is within tolerance in the Y-axis the 
vertical lines in the translational cue will become 
bolder. And finally, when the POR is within the limit 
in the Z-axis, the horizontal lines of the translational 
cue become bold. Once all of the lines which comprise 
the translational cue are bold, the operator will know 
that the manipulator tip is within tolerance in the X, Y, 
and Z axes. 

For the rotational cues in the R/T Submode, the 
circle becomes bold when the manipulator’s POR is 
within the yaw limit. The horizontal line drawn through 
the circle is made bold when the pitch limit is satisfied. 
And the vertical roll indicator is made bold when the 
POR is within the roll limit. As with the translational 
cue, when the manipulator POR is within limit in yaw, 
pitch, and roll the entire rotational cue will be bold. 
Fig. 4 shows an example of the bold feature indicating 
that the X-axis and the yaw axes are within range. The 
tolerances can be set to different values for each degree 
of freedom. This feature is also implemented in the 
MPD Pilot Submode. 


2.4 Advanced Features of the MPD 

As a result of the experiments described above and 
comments from astronauts, mission designers, and 
astronaut trainers, a number of recommendations for 
improving the MPD were gathered. These 
recommendations have resulted in the implementation 
of a number of new features. The following section 
describes each of the new features and their benefits. 

2.4.1 Highlighting 

One advanced feature is highlighting cues to help 
the operator distinguish between the lines which 
represent the rotational and translational cues, and the 
stationary reference line at the center of the screen. 
This feature becomes most useful when the manipulator 
is reaching its target position and the operator is trying 
to align the cues to the stationary reference line. This is 
one of the most critical phases of any operation. 
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Fig. 4. MPD highlighting feature. 


For each task there are defined tolerance limits, for 
each degree of freedom, within which the manipulator 
is considered to be at its desired final position. Based 
on this information a highlighting feature was 
implemented which indicates to the operator when the 
manipulator is within the defined limit for each degree 
of freedom. This indication is achieved by increasing 
the width of specific lines on the rotational and 
translational cues when the manipulator position and 
attitude are within the specified range. For example, 
when the Pbint of Resolution (POR) of the manipulator 
is within the specified range in the X-axis (see figure 6- 
8) the square, in the R/T Submode, will become bolder 


Another benefit of using line width as an indication 
of reaching final position is the ability to reach the 
desired position in any one axis regardless of where the 
cues are on the screen with respect to the reference line. 
For instance, in the event that a particular translational 
axis needs to be aligned befo re the other axes this can 
be done without the translational cue being lined up 
with the stationary reference line. This occurs when a 
payload must be centered in the X and Y axes before 
being lowered into the shuttle bay. During such a task 
the operator would have to adjust the size of the square 
to coincide with the length of the reference line without 
having the translational cue over the reference line. 
Without the added feature this would be accomplished 
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by referring to the deltas being displayed on the upper 
left-hand comer, recalling the defined limit for each 
axis, and watching the translational cue. With the 
aHAvi feature the operator need only concentrate on the 
translational cue receiving a visual signal when the 
POR is within range for the desired axis( in this 
example the X-axis). 

2 4 2 Color Cues 

In addition to the highlighting feature, the MPD 
display now provides color cues to help distinguish 
between the translational and rotational cues, and the 
stationary reference line. The use of color is useful 
when the manipulator POR is close to its final 
destination as shown in Fig 5. As can be seen in the 
figure it can be difficult to differentiate between the 
translational cue, rotational cue, and the reference line. 
In the current MPD implementation the translational 
cue is drawn in red, the rotational in green and the 
reference line in white. 
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Fig. 5. MPD Display need for color cues. 

Color cues are also being considered in conjunction 
with the highlighting feature to give the operator 
information on the proximity to the final destination. 
The idea is to define a range, like the limits described in 
the previous section, which when entered by the 
mani pulator POR would cause the translational and 
rotational cues to change color. This would supply the 
operator with a visual cue that the manipulator POR is 
reaching its destination and in turn the hand controller 
inputs should be reduced in order avoid going beyond 
the desired final position. Once the previously 
drarihfri final limits are reached, the translational and 
rotational cues’ colors can again be changed as the lines 
get bold. In this way the operator is given two signals 


tha t the manipulator has reached the final POR, bolder 
lines and change in color . 9 

2.4.3 Direction Cues 

In the original implementation of the MPD, the 
deltas between the current and final POR positions were 
displayed as signed numbers in the upper left-hand 
comer of the screen. The sign of the numbers is 
provided as an indication of the direction in which the 
delta exists. In Fig. 5 this can be seen in the Z-axis and 
pitch digital delta readouts. This approach required the 
operator to mentally transform the sign cue to the 
coordinate frame in which they are working, then figure 
out the corresponding hand controller deflections 
required to compensate for the deviation. However, 
what usually occurs is that the operator inputs the 
wrong direction based on the sign delta. 

To alleviate this difficulty the MPD includes a 
feature referred to as “Direction Cues". Direction Cues 
supply the operator with instructions of the necessary 
hand controller deflections to remove the deltas in each 
degree of freedom. The Direction Cues can be seen in 
Fig. 5 as letters following the deltas in the upper left- 
hand comer of the display. The letters I or O are used 
to indicate in or out deflection of the translational hand 
controller, L or R for left or right deflection of the 
translational hand controller, and U or D for up or down 
deflection of the translational hand controller. For the 
rotational Direction Cues the letters U, D, L , and R are 
used in the way as with the translational Direction 
Cues. Fig. 5 shows the display signaling the operator to 
deflect the translational hand controller out, left, and 
down and the rotational hand controller down, left, and 
left for the pitch, yaw, and roll axes respectively. With 
the addition of Direction Cues the operator is presented 
with straight forward indications of the necessary hand 
controller deflections eliminating the possibility of 
unnecessary and potentially dangerous movement of the 
manipulator. 

2.4. 4 Flv-TofFlv-From Option 

The original version of the MPD displays used 
what is referred to as “fly- from,” or outside-in, 
convention to show the deviation between the current 
manipulator POR position and the desired final 
position. In the fly-from convention the objective is to 
input the necessary hand controller deflections to move 
the graphical cues from their current positions to a 
specified reference point in the display. In the MPD 
displays the reference point is the stationary reference 
line in the center of the screen. As operators with 
varying backgrounds used the MPD displays two points 
were made about the utilization of the fly-from 
convention. 
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First, it was not obvious from tbe information 
presented by tbe MPD displays that a fly-from 
convention was being used. And secondly, not 
everyone is used to tbe fly-from convention. Some 
operators are more comfortable with the “fly-to”, or 
inside-out, convention. In the fly-to convention the 
objective is to deflect the hand controllers in such a way 
as to move a specified reference point, the stationary 
reference line, to the current position of the graphical 
cues. As the hand controller inputs are generated the 
graphical cues move towards the reference line giving 
tbe illusion that tbe reference line is moving. 10 

Having reached the conclusion that neither one of 
the conventions exhibit any inherent advantages, the 
MPD display now gives tbe operator a choice of using 
either option. At the beginning of each task the 
operator selects whether tbe graphical cues are shown in 
the fly-to or fly-from convention. Once this selection is 
made, the MPD displays the option in tbe top center 
part of the screen as can be seen in Fig. 5. This new 
feature gives the flexibility to use the display in tbe 
convention which is most comfortable to the operator 
and also makes the current selection obvious at all 
times. 

2.4.5 Coordinate Frame Selection 

The last addition to the original MPD display is tbe 
capability to select between the different coordinate 
frames in which to command the manipulator POR. 
Originally the commands where all based in the orbiter 
coordinate frame which is shown in Fig. 6. 


Yaw 


Z 

Fig 6. Space Shuttle coordinate reference frame. 

With the addition of the coordinate frame selection 
feature the operator now has a choice between orbiter, 




end effector, and payload coordinate frames. In the 
case of the space shuttle, this is a major improvement 
over the information currently displayed in the aft flight 
deck which is always in orbiter reference mode. An 
example of an end effector coordinate frame is depicted 
in Fig. 7. 


Y 



Fig. 7. End effector coordinate frame. 

The payload coordinate frame is different for each 
payload and can sometimes coincide with either the end 
effector or orbiter coordinate frames. Fig. 8 shows an 
example of a payload coordinate frame. 
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The coordinate frame selection feature provides 
consistency in the way the graphical cues display 
changes in the different axes. For example, in the R/T 
Submode movement in the X-axis is always depicted as 
changes in the size of the square of the translational 
cue. Motion in the Y-axis is always shown as a change 
in the translational cue’s horizontal position on the 
screen. And motion in the Z-axis is always shown as a 
change in the translational cue’s vertical position on the 
screen. The selected reference frame is displayed in the 
top center part of the screen (see Fig. 5). 1 1 

1 Joint Ang le Display Mode 



Fig. 10. Nominal operations Joint Angle Display 


The second major mode of the MMDS is the Joint 
Angle Display (JAD) Mode. The JAD is comprised of 
a set of bargraphs which represent the position of each 
joint of a manipulator. The JAD mode has three 
submodes: 1) nominal operations, 2) joint limits, and 3) 
single joint operations. 

3 1 Nominal Operations Submode 

The nominal operations mode displays the current 
joint positions to the operator. For example, the SRMS 
has six joints as is shown in Fig. 9. Fig. 10 shows how 
the six joint values for the SRMS would be presented to 
the operator. Note that each joint in Fig. 9 is listed in 
Fig. 10. Parh bar graph represents the current joint 
angle. The bar graphs are updated in real-time based on 
the changing encoder values at each joint 


3 2 Joint Li mits Submode 

The second submode of the display will include all 
the features of the first submode plus cues to indicate 
the location of the joint limitations. As can be seen in 
Fig. 11 the joint limits are depicted by the small 
triangles to the right of each bar graph. For instance 
Fig. 1 1 shows that for the SY joint the joint limits are at 
±180'. This display could also emit an audible tone 
when any joint reaches a limit. By including the 
audible tone the operator will be notified of a joint limit 
error without having to constantly monitor each joint. 
Having the features designed in this submode of the 
JAD provides the operator with a tool to avoid joint 
limits. 


Wrist 

Roll 



Shoulder 
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Fig. 9 SRMS Manipulator 
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Fig. 11. Joint Limits in the Joint Angle Display 


13 Single Joint Operations Submode 


Another application for the JAD will be single joint 
operations when the operator needs to drive the arm one 
joint at a time. This operational scenario occurs on the 
space shuttle during failure modes which make 
controlling all joints concurrently impossible (for 
example, a hand controller failure). During these 
operations, the Single Joint Operations submode will 
not only provide the operator with information on the 
current joint positions and joint limits, but will also 
provide the operator with operational cues. These cues 
will include the amount of deflection needed for each 
joint, and the joint sequence. One limitation of this 
display is, however, that the encoder data from the 
manipulator joints are needed to run the display and 
might not be available in the event of a failure. 


sv sp cp uiv uir u tn 



Fig. 12. Wrist pitch joint indication. 

Fig. 12 provides an example of the Single Joint 
Operations Submode display. Fig. 12 indicates that the 
Wrist Pitch joint should be moved to -86 degrees. Once 
the operation in Fig. 12 is complete, the next step would 
be displayed 


With the completely integrated MMDS the 
operator is supplied with a complete, concise, and 
flexible view of the state of the manipulator at all times. 
This complete view includes information on both the 
manipulator POR position through the use of the MPD 
displays, and the position of each individual joint 
through the use of the JAD. Using the MMDS, a 
typical grapple and unberth task with SRMS can be 
described as follows. 

The operator begins the task using the MPD 
display of their choice, Pilot or RotationaLTranslational 
Submode, in end effector coordinate reference frame 
and fly-from mode. As the operator maneuvers toward 
the grapple fixture, they can at any time switch to the 
JAD viewing the status of each joint and their proximity 
to any limits. Once the POR is within the predefined 
limits the translational and rotational cues are 
highlighted. At this time the payload is grappled and 
the operator switches to orbiter coordinate reference 
frame. 

With the payload grappled a new target POR 
position is entered and the translational and rotational 
cues adjust to show the new deltas. The operator now 
begins to issue the appropriate hand controller 
deflections to move the manipulator towards the new 
destination. If at any point during the task a joint limit 
is reached, the JAD will sound an audible tone 
anunciating that such a limit has been reached. 

Upon recognizing the joint limit alarm, the operator 
will switch to the JAD where he/she can rapidly 
identify the errant joint. The operator would then 
switch to single joint mode and command the wayward 
joint away from the limit using the JAD. Once the joint 
is backed away from the limit the operator can revert to 
the MPD display to reach the final POR. 


Another task would be to berth the payload into the 
orbiter bay. Once again the new target position is 
entered and the translational and rotational cues 
adjusted to show the deltas. At this point the operator 
can use the payload coordinate reference frame to drive 
the payload into its berthed position. Once the final 
berthed position is reached the task is completed. 

One final note with respect to the flexibility of the 
MMDS. At any time during the task described above 
the operator hais the capability to choose between how 
and what information is displayed without having to 
restart the MMDS. The operator can switch between 
the MPD or JAD, Pilot or RJT Submode, and coordinate 
reference frames. This capability gives the operator the 
ability to command the manipulator in a way that is 
most suitable to their background and training. 
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5. Conclusions 

Based on the development and experimental results 
presented in this paper, the MMDS can be expected to 
provide significant operational benefits including 
providing the operator with useful manipulator position 
information, reducing control problems associated with 
the poor viewing conditions, reducing operator 
workload, reducing training time, and assisting the 
operator with performing unscheduled or unpracticed 
procedures. The MMDS has space based application 
for the SSRMS space station as well as for ground 
control of space based manipulators. Its potential 
application areas will hopefully be expanded into 
environmental, hazardous waste, nuclear, and undersea 
remote manipulation environments. 
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